System and method for controlling audio output associated with haptic effects

ABSTRACT

An apparatus, a processor-readable medium, and a method are provided that are configured to cause a haptic effect and an audio effect to be output substantially concurrently. The haptic effect has a frequency and the audio effect has a frequency different from the frequency of the haptic effect. At least one of the frequency of the haptic effect and the frequency of the audio effect is varied while maintaining substantially constant an average energy of the haptic effect. Varying the frequency of the audio effect can cause a perceived frequency of the haptic effect to change.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 U.S.C. 119(e) of U.S.Provisional Application Ser. No. 60/590,432, entitled “System and Methodfor Controlling Audio Output Associated with Haptic Effects,” filed Jul.23, 2004, which is incorporated herein by reference in its entirety

BACKGROUND

The invention relates generally to haptic feedback devices. Morespecifically, the invention relates to a system and method forcontrolling audio output associated with haptic effects.

Devices that provide haptics, such as tactile feedback, have enjoyedincreased popularity in recent years. These devices are used in avariety of different applications. For example, devices providinghaptics are popular in various applications, where the haptic feedbackenhances the overall gaming experience of a user. For example,haptic-enabled controllers, such as mouse devices, can be configured toprovide haptic feedback to a user while the user interacts with anoperating system (OS), or other application.

Existing devices, however, do not effectively control audio outputassociated with haptic feedback. Accordingly, it would be desirable tocontrol effectively audio output associated with haptic effects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a processor system, according to anembodiment of the invention.

FIG. 2 is a diagram illustrating a haptic device, a controller, and asensor, according to an embodiment of the invention.

FIG. 3 is a block diagram of a haptic device, according to an embodimentof the invention.

FIG. 4 is a diagram of multiple frequency ranges of haptic effectsoutput by a haptic device, according to an embodiment of the invention.

FIG. 5 is a plot of a magnitude versus frequency response of a hapticdevice, according to an embodiment of the invention.

FIG. 6 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention.

FIG. 7 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention.

FIG. 8 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention.

FIG. 9 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention

FIG. 10 is a diagram of linearization of voltages of a haptic device,according to an embodiment of the invention.

FIG. 11 is a diagram of various parameters associated with a smootheffect according to an embodiment of the invention.

FIG. 12 is a diagram of various parameters associated with a strongeffect according to an embodiment of the invention.

FIG. 13 is a diagram of various parameters associated with a sharpeffect according to an embodiment of the invention.

DESCRIPTION

A system and method for controlling audio output associated with hapticeffects are described. More specifically, audio output associated withhaptic effects can be controlled to modify a perceived experience of thehaptic effects. For example, by modifying the audio output, a user canbe made to perceive a frequency of a haptic effect as being differentthan the actual frequency.

According to one or more embodiments of the invention, control signalscan be configured to cause haptic effects to be output across a widerange of frequencies. These control signals can independently controlhaptic effects within any frequency range from among multiple frequencyranges. This can occur, for example, using either a single controller ormultiple controllers configured to output control signals from eachfrequency range. For example, a single controller can output controlsignals that independently control haptic effects in each of multiplefrequency ranges. Alternatively, multiple controllers can be used, suchthat each controller outputs control signals within a single frequencyrange of multiple frequency ranges, each controller being uniquelyassociated with each frequency range.

Audio output associated with a haptic effect is generated in at leastone frequency range of multiple frequency ranges when that haptic effectis output in response to a corresponding control signal. For example,when a haptic effect is output by a haptic device in response to acontrol signal, the haptic device can also create audible sound or, inother words, an audio output based on the movement of the haptic device.The audio signal heard by a user can correspond to a frequency of ahaptic effect that is beyond the tactile detection capabilities of theuser. In other words, although a user cannot feel a difference in thefrequency of a haptic effect above a certain threshold frequency, theuser can hear audio associated with such higher frequencies. Thus,although varying such tactile-imperceptible frequencies will not cause auser to feel a difference in a frequency of a haptic effect, the userwill be able to hear such a variation. Because a user can hear anincrease or decrease in frequency of the audio output, the user willperceive that the haptic effect has changed, and in many cases willbelieve that he or she has felt the change in the overall experience.

One or more embodiments of the invention provide an extended perceivedfrequency range of haptic effects. More specifically, in addition to therange of haptic effects that can be tactilely detected by the user, arange of effects that are detected audibly by a user can be can be addedsuch that the perceived overall experience has a greater frequencyrange. Because a user is able to sense an increased range offrequencies, more information can be communicated to the user using suchcombination of haptic effects and audio output.

When pulse-like control signals are used to generate haptic effects,control signals having a constant average energy can be used to providea variety of different audio output frequencies. The different audiooutput frequencies can cause a user to believe that he or she is sensingtactilely a different frequency of a haptic effect, even thoughtactilely sensing such a difference would not be possible. Thus,according to one or more embodiments of the invention, a variety ofoverall experiences (each having a haptic component and an audiocomponent) caused by a corresponding variety of control signals, each ofwhich has the same average energy, is able to be sensed by a user via acombination of the haptic effect and the associated audio output, eventhough the variety of haptic effects alone would be perceived as havingthe same feel without the audio output. The average energy can bemaintained constant by varying the frequency and/or duty cycle of acontrol signal inversely with the magnitude of a control signal. Thus,as the frequency of the control signal is increased, the magnitudedecreases and, conversely, as the frequency of the control signaldecreases, the magnitude increases, to maintain a constant averageenergy of the carrier signal.

FIG. 1 is a block diagram of a processor system, according to anembodiment of the invention. The processor system 110 illustrated inFIG. 1 can be, for example, a commercially available personal computer,portable electronic device, or a less complex computing or processingdevice (e.g., a device that is dedicated to performing one or morespecific tasks). For example, the processor system can be a mobiletelephone, a PDA, a portable gaming system, an MP3 player, or the like.Alternatively, the processor system 110 can be a terminal dedicated toproviding an interactive virtual reality environment, such as a gamingsystem, or the like. Although each component of the processor system 110is shown as being a single component in FIG. 1, the processor system 110can include multiple numbers of any components illustrated in FIG. 1.Additionally, multiple components of the processor system 1 10 can becombined as a single component.

The processor system 110 includes a processor 112, which according toone or more embodiments of the invention, can be a commerciallyavailable microprocessor capable of performing general processingoperations. Alternatively, the processor 112 can be anapplication-specific integrated circuit (ASIC) or a combination ofASICs, which is designed to achieve one or more specific functions, orenable one or more specific devices or applications. In yet anotheralternative, the processor 112 can be an analog or digital circuit, or acombination of multiple circuits.

Alternatively, the processor 112 can optionally include one or moreindividual sub-processors or coprocessors. For example, the processorcan include a graphics coprocessor that is capable of renderinggraphics, a math coprocessor that is capable of efficiently performingcomplex calculations, a controller that is capable of controlling one ormore devices, a sensor interface that is capable of receiving sensoryinput from one or more sensing devices, and so forth.

The processor system 110 can also include a memory component 114. Asshown in FIG. 1, the memory component 114 can include one or more typesof memory. For example, the memory component 114 can include a read onlymemory (ROM) component 114 a and a random access memory (RAM) component114 b. The memory component 114 can also include other types of memorynot illustrated in FIG. 1 that are suitable for storing data in a formretrievable by the processor 112. For example, electronicallyprogrammable read only memory (EPROM), erasable electricallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component114. The processor system 110 can also include a variety of othercomponents, depending upon the desired functionality of the processorsystem 110. The processor 112 is in communication with the memorycomponent 114, and can store data in the memory component 114 orretrieve data previously stored in the memory component 114.

The processor system 110 can also include a haptic device 116, which iscapable of providing a variety of haptic output. For example, the hapticdevice 116 can be configured to output basis haptic effects, such asperiodic effects, magnitude-sweep effects, or timeline haptic effects,each of which is described in greater detail below. According to one ormore embodiments of the invention, the haptic device 116 can include oneor more force-applying mechanisms, which are capable of outputtinghaptic effects or force, to a user of the processor system 110 (e.g.,via the housing of the processor system 110). These effects or forcescan be transmitted, for example, in the form of vibrational movementcaused by the haptic device 116 (e.g., caused by a rotating mass, apiezo-electric device, or other vibrating actuator), or in the form ofresistive force caused by the haptic device 116.

The processor system 110 can also, according to one or more embodimentsof the invention, include a sensor 118 that is capable of receivinginput from a user, the haptic device 116, or is otherwise capable ofsensing one or more physical parameters. For example, according to oneor more embodiments of the invention, a sensor 118 can be configured tomeasure speed, intensity, acceleration, or other parameters associatedwith a haptic effect output by the haptic device 116. Similarly, thesensor 118 can be configured to sense environmental or ambientconditions of the processor system's surroundings. The sensor 118 caninterface and communicate with the processor 112 by way of a sensorinterface (not shown) within the processor 112.

The processor system 110 can also include a controller 120, which canoptionally be internal to the processor 112, or external thereto, asshown in FIG. 1. The controller 120 can be configured to control thehaptic device 116 when the processor 112 is not directly controlling thehaptic device 116. Similarly, the controller 120 can control the memory114 and/or the sensor 118, as well as devices external to the processorsystem 110 by way of an input/output (I/O) component 124 (describedbelow).

The various components of the processor system 110 can communicate withone another via a bus 122, which is capable of carrying instructionsfrom the processor 112 and/or the controller 120 to other components,and which is capable of carrying data between the various components ofthe processor system 110. Additionally, signals received via the sensor118 can be communicated to the processor 112 or the controller 120 byway of the bus 122. Data retrieved from or written to memory 114 can becarried by the bus 122, as are instructions to the haptic device 116.Instructions to the haptic device 116 can be provided in the form ofhaptic-effect signals (e.g., basis haptic-effect signals), for example,which can be provided by the processor 112, the controller 120, ordevices external to the processor system 110.

The components of the processor system 110 can communicate with devicesexternal to the processor system 110 by way of an input/output (I/O)component 124 (accessed via the bus 122). According to one or moreembodiments of the invention, the I/O component 124 can include avariety of suitable communication interfaces. For example, the I/Ocomponent 124 can include, for example, wireless connections, such asinfrared ports, optical ports, Bluetooth wireless ports, wireless LANports, or the like. Additionally, the I/O component 124 can include,wired connections, such as standard serial ports, parallel ports,universal serial bus (USB) ports, S-video ports, large area network(LAN) ports, small computer system interface (SCSI) ports, and so forth.

FIG. 2 is a diagram illustrating a haptic device, a controller, and asensor, according to an embodiment of the invention. FIG. 2 also showsdata values provided to the system (e.g., user input 202 and controlparameters 204). The elements shown in FIG. 2 can be used with theprocessor system 110, or a similar device.

The controller 120 is configured to output control signals that areconfigured to cause haptic effects to be output by the haptic device116. As shown in FIG. 2, user input 202 can optionally be provided(e.g., via the I/O component 124 shown in FIG. 1) and/or received by anoptional sensor 118. The user input 202 can also optionally be provideddirectly to a controller 120 (e.g., by way of the sensor 118, or someother devices configured to accept and convey user input). The sensor118 can also optionally receive information from the haptic device 116.For example, the sensor 118 can sense the actual movements of the hapticdevice 116.

According to an arrangement of the system shown in FIG. 2, thecontroller 120 can optionally receive data from the sensor 118, and canoptionally receive user input 202 and control parameters 204. Based onany data received from the sensor 118, any received user input 202,and/or any received control parameters 204, the controller 120 controlsthe haptic output of the haptic device 116 (e.g., the controller 120sends control signals configured to cause haptic effects). For example,the controller 120 can execute a feedback algorithm, controlling thehaptic device 116 based on feedback received from the haptic device 116.The controller 120 controls the output of the haptic device 116 by acontrol signal that the controller 120 outputs to the haptic device 116.

The control signal output by the controller 120 can be based on a numberof parameters, including, for example, control parameters 204. Forexample, control parameters 204 and other parameters that can be used bythe controller 120 to control the haptic device 116 can be stored in thememory component 114 of the processor system 110, or by another suitablememory component. For example, the control parameters 204 can includeinput from an electronic system, a portable gaming device, a cellulartelephone, or the like. According to one or more embodiments of theinvention, the controller receives control parameters (e.g., gamingdevice input, cellular telephone input, etc.), and does not include asensor. According to such embodiments, user input can optionally bereceived directly by the controller, or can be omitted entirely,depending upon the desired function of the system in which thecontroller is used.

According to one or more embodiments of the invention, the system shownin FIG. 2 can be used in a stand-alone device, such as a mobiletelephone, portable electronic device (e.g., a PDA, etc.), or otherdevice. In a mobile telephone embodiment, for example, haptic output canbe provided in the form of haptic effects via the haptic device 116 inresponse to status events (e.g., a message received signal, a networkindicator signal, etc.), user input (e.g., mode changes, keypad dialing,option selections, etc.), incoming calls, or other events.Alternatively, the system shown in FIG. 2 can be used in a configurationwhere a processor, such as the processor 112 of the processor system 110shown in FIG. 1, can be connected to an external device, and theprocessing tasks can be divided among the devices, as desired.

The controller 120 can generate a variety of different control signalsto drive the haptic device 116, several of which will be described ingreater detail below. For example, the controller 120 can send a controlsignal to the haptic device 116, which is configured to cause the hapticdevice 116 to output a corresponding haptic effect. Examples of suchcontrol signals include, pulse width modulation (PWM) signals (e.g.,pulse signals having a given duty cycle), sinusoidal signals, and otherperiodic signals (e.g., triangle waves, square waves, etc.).Additionally, the controller 120 can modulate control signals using oneor more haptic envelopes.

The controller 120 also can be configured to provide a lead-in pulse atthe beginning of a control signal, and/or a braking pulse, at the end ofa control signal, which are configured to decrease response time of thehaptic device 116. For example, the lead-in signal reduces the time forthe haptic device 116 to initiate outputting a haptic effect associatedwith the control signal. The braking pulse, on the other hand, decreasesthe time it takes for the haptic device 116 to cease a haptic effectcurrently being output. In addition to signals described above, such asperiodic signals, the controller 120 can output a variety of othercontrol signals, such as non-periodic signals, that are configured tocause the haptic device 116 to output haptic effects.

FIG. 3 is a block diagram of a haptic device 116 shown in FIGS. 1 and 2.As shown in FIG. 3, the haptic device 116 includes an actuator 302, anelastic member 304 and a mass 306. The haptic device 116 is configuredto provide haptic feedback. The actuator 302 is operably connected tothe elastic member 304, and the elastic member 304 is operably connectedto the mass 306. The actuator 302 can include, for example, a motor(e.g., a brush motor, a brushless motor, etc.). The elastic member canprovide a desired amount of coupling rigidity between the actuator andthe mass 306.

When control signals are received by the haptic device 116, the actuator302 provides force to the elastic member 304. Some of the force appliedto the elastic member 304 is translated to the mass 306, and causes themass 306 to move. By causing the mass 306 to move, haptic effectscommanded by the control signals are output by the haptic device, andcan be output to a user. The actuator 302 can be configured, forexample, to cause the mass to rotate in response to the control signalsreceived by the haptic device. Alternatively, the actuator can move themass 306 in other directions (e.g., vibrating the mass, moving the masslaterally, etc.).

The configuration shown in FIG. 3 is only one example of a configurationof a haptic device 116. Other configurations that vary from theconfiguration shown in FIG. 3 can be used as a haptic device 116,according to one or more embodiments of the invention. For example, theelastic member 304 can be coupled to the mass 306 by a flexiblecoupling; the elastic member 304 can be coupled to the actuator 302 by aflexible coupling. In an alternative embodiment, the elastic member canbe coupled between actuator and a mechanical ground, and the actuatorcan be directly coupled to the actuator. Examples of haptic devices thatcan be used in connection with one or more embodiments of the inventioninclude an eccentric-rotating-mass (ERM) haptic device and a harmonicERM (HERM) haptic device, described in detail in copending U.S. patentapplication Ser. No. 10/301,809, which is incorporated by referenceherein in its entirety.

FIG. 4 is a diagram of multiple frequency ranges of haptic effects thatcan be output by a haptic device 116, according to an embodiment of theinvention. A low-frequency range extends from approximately DC (i.e., 0Hz) to a low-frequency limit f_(L), which can vary depending upon thecontrol signal being used to cause a haptic effect and the desiredcharacteristics of the haptic effect. A mid-frequency range extends fromthe low-frequency threshold frequency f_(L) to a high-frequencythreshold frequency f_(H), which can vary depending upon the controlsignal being used to cause a haptic effect and the desiredcharacteristics of the haptic effect. A high-frequency range extendsfrom the high-frequency threshold frequency f_(H) to all higherfrequencies.

According to one or more embodiments of the invention, at least onefrequency range from the frequency ranges shown in FIG. 4 can have anaudio output associated with the haptic effect. The audio output canoccur for haptic effects having a frequency within the at least onefrequency range or for haptic effects having a frequency beyond the atleast one frequency range, depending upon the desired performance of thesystem.

For example, according to one or more embodiments of the invention, themid-frequency range shown in FIG. 4 can have an audio output associatedwith a haptic effect having a frequency within the mid-frequency range.The haptic effect having a frequency within the mid-frequency range canbe varied in a manner such that the associated audio output varies,while any changes in the frequency of the haptic effect remainstactilely undetectable to a user. Because the audio output varies (e.g.,changes frequency of the audio output), a user aurally detects thechange in the audio output, and believes that he or she has tactilelydetected a change in the haptic effect. Said another way, by varying theaudio output, the user may perceive that the overall effect (thecombination of a haptic effect and an audio effect) has changed andattribute such a change, at least in part, to the user's tactileexperience. Additionally, in one or more embodiments, the average energyof a control signal used to cause the haptic effect to be output can bemaintained substantially the same while the associated audio output isvaried, causing a user to detect an increase in the audio output andbelieve that he or she has tactilely perceived a change in the hapticeffect.

Although FIG. 4 illustrates only three frequency ranges, the number offrequency ranges used according to one or more embodiments of theinvention can be varied. For example, many more frequency ranges can beused, among which multiple frequency ranges can include an audio outputassociated with the haptic effects having frequencies within thosefrequency ranges.

Haptic effects having frequencies within each of the frequency rangesshown in FIG. 4 can be separately controlled. This can occur, forexample, using a single controller, that separately controls the hapticeffects associated with each of the frequency ranges shown in FIG. 4.Alternatively, each frequency range shown in FIG. 4 can have a uniquelyassociated controller, which is configured to control haptic effectshaving frequencies within that frequency range.

FIG. 5 is a plot of a magnitude versus frequency of a haptic effect,according to an embodiment of the invention. The plot shown in FIG. 5 isnot drawn to scale, and is intended only as an example of thecorrelation between the magnitude and frequency of a haptic effect, andhow a user perceives them. The magnitude versus frequency response shownin FIG. 5 indicates that, as the frequency of a haptic effect isincreased, the magnitude of that haptic effect appears also to increaseto a user. A first portion 510 of the curve shown in FIG. 5 represents aregion of haptic effect frequencies within which a user can detectchanges in frequency. Within some region of frequencies (e.g., beginningnear the high-frequency threshold frequency f_(H) in the plot of FIG.5), referred to as a diminished-sensitivity region, a user perceivessome increases in the frequency of a haptic effect as increases inmagnitude (and not the frequency) without being able to detect tactilelythe increases in frequency. A second portion 520 of the curve shown inFIG. 5 represents this region, where the user has difficulty tactilelydetecting changes in frequency. Near where the diminished-sensitivityregion begins (i.e., the area of intersection of the first portion 510and the second portion 520 of the curve shown in FIG. 5), a user'sperception of increasing frequencies of combination of a haptic effectand audio output is illustrated using a line 520. Similar lines can bedrawn to illustrate a user's perception of increasing frequencies of acombination of a haptic effect and an audio output is generated by thehaptic device.

According to one or more embodiments of the invention, a pulse-like,periodic control signal is configured to cause the haptic effects to beoutput having frequencies within each of the ranges shown in FIG. 4.Examples of such signals are discussed in greater detail below.Generally speaking, the period between pulse features of the controlsignal corresponds to a low-frequency component (e.g., a hapticenvelope) of a haptic effect at lower frequencies. At lower frequencies,it is these low-frequency components (perceived as pulses) that are mosteasily tactilely detected by a user. As the period between the pulses ofthe control signal decreases (i.e., the frequency of the pulsesincreases), the haptic device 116 reaches a state where it is movingalmost the entire period, even during the portions of the period when nopulse in the control signal exists. Over increasing control signalfrequencies where this begins to occur, the haptic device 116 is said tobe operating in “saturation mode.” For example, in the case of arotating-mass device, when the haptic device 116 reaches the saturationmode, despite the fact that the control signal pulses are notcontinuously on and, therefore, are causing low-frequency components inthe haptic effect, the mass of the device continues to rotate. Thesaturation mode may or may not correspond to the diminished-sensitivityregion, depending upon the physical characteristics of the haptic device116 or other parameters.

When an audio output associated with a haptic effect is output at thesame time as the haptic effect, a user perceives the frequency of thehaptic effect to increase due to an increase in the frequency of theaudio output. This is illustrated, for example, by a line 530 extendingfrom the magnitude versus frequency curve shown in FIG. 5. Thisperceived change in frequency of the haptic effect due to the audiooutput can occur, for example, at the beginning of thediminished-sensitivity region (i.e., where it begins to be difficult fora user to tactilely detect variations in frequency). According to one ormore embodiments of the invention, the perceived increased frequency ofthe haptic effect occurs when the haptic device is being driven withinthe mid-frequency range (i.e., a frequency between f_(L) and f_(H)), asshown in FIG. 5. Using the audio output to increase the frequency rangea user perceives a haptic device to have allows a user to experience anincreased perceived frequency range in the overall experience, andspecifically the perceived haptic effect, without being limited by theperformance range of the haptic device. Although not shown, multiplelines similar to the illustrated line 530 can be used to represent anaudio output changing the frequency that a user perceives either withinor outside of the diminished-sensitivity region. Also, although the line530 representing a frequency perceived by a user indicates an essentialconstant perceived magnitude, it also is possible to change themagnitude perceived by a user, depending upon the audio output that isproduced.

Several signals are described below in greater detail. These signals areonly examples, however, and it should be recognized that there are manyother signals that are suitable for acting as control signals, dependingupon the desired haptic effects to be output and audio output to beproduced. Examples of control signals that can be used in connectionwith one or more embodiments of the invention are described in detail incopending U.S. patent application Ser. Nos. 09/669,029 and 10/671,465,each of which is incorporated by reference herein in its entirety.Similarly, other control signals, as well as haptic devices that can beused in connection with one or more embodiments of the invention aredescribed in detail in U.S. Pat. No. 6,275,213, which is incorporated byreference herein in its entirety.

FIG. 6 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention. The control signal shown inFIG. 6 can be used to closely control the frequency of a haptic effect.The control signal shown in FIG. 6 is a series of pulses, each having apositive on-time. The pulses are periodic, having a period thatcorresponds to the desired frequency of the haptic effect to be output,as defined by Equation 1 below: $\begin{matrix}{{T = \frac{1}{f_{D}}},} & (1)\end{matrix}$where T is the period of the control signal (i.e., the time periodbetween two adjacent pulses of the control signal), and f_(D) is thedesired frequency of the output of the haptic effect.

Changes in magnitude of a haptic effect caused by the control signalshown in FIG. 6 can be conveyed by proportional changes of magnitude ofthe pulses of that control signal. A change in magnitude of the outputhaptic effect that is based on the control signal shown in FIG. 6 variesproportionately to the change in magnitude of the control signal. Theduration of the pulses (i.e., the positive on-time or duty cycle) can beselected according to the values shown in Table 1 below to provide adistinct frequency pattern, depending upon the frequency range of thehaptic effect being selected. TABLE 1 Duty cycle/on-time values forcontrol signal of FIG. 6 Frequency Range Frequencies Duty Cycle/On-TimeLow f_(D) ≦ 6.66 Hz 75 ms Mid 6.66 Hz < f_(D) ≦ 10 Hz 50% High 10 Hz <f_(D) ≦ 100 Hz  50% @ 10 Hz f_(D) > 100 H 100% @ 100 Hz and above

The frequency ranges shown in Table 1 above can correspond to the threeranges shown in FIG. 4 (with “other” being included in thehigh-frequency range), according to one or more embodiments of theinvention. To achieve higher desired frequencies f_(D) of a hapticeffect, the duty cycle of the control signal is increased. For example,in the transition from 10 Hz to 100 Hz in the high-frequency range, theduty cycle increases from 50% to 100%. This increase in duty cycle canbe a linearly increase, or another type of increase, if desired.

The duration of the pulses (i.e., the positive on-time, or duty cycle)can alternatively be selected according to the values shown in Table 2below to provide a strong haptic effect magnitude, depending upon thefrequency range of the haptic effect being selected. TABLE 2 Dutycycle/on-time values for control signal of FIG. 6 Frequency RangeFrequencies Duty Cycle/On-Time Low f_(D) ≦ 10 Hz 75 ms Mid 10 Hz < f_(D)≦ 16 Hz 75% High 16 Hz < f_(D) ≦ 100 Hz  50% @ 10 Hz f_(D) > 100 Hz 100%@ 100 Hz and above

The frequency ranges shown in Table 2 above can also correspond to thethree ranges shown in FIG. 4 (with “other” being included in thehigh-frequency range), according to one or more embodiments of theinvention. To achieve greater magnitude of a haptic effect, the lengthof the duty cycle of the control signal is increased in thehigh-frequency range. As discussed above, the duty cycle can beincreased linearly, or in some other desirable manner.

FIG. 7 is a diagram of a control signal used to control a haptic device,according to another embodiment of the invention. The control signalshown in FIG. 7 is a bi-directional control signal that includesmultiple bi-directional pulses, and is configured to create a hapticeffect with a strong magnitude. These bi-directional pulses areperiodic, and have a period corresponding to the desired frequency f_(D)of the haptic effect to be output (defined by Equation 1 above).Seventy-five percent of the bi-directional pulse is a positive pulseportion, and twenty-five percent of the bi-directional pulse is anegative pulse portion. Values associated with the pulse size forvarious frequency ranges are shown below in Table 3. TABLE 3 Duty cyclevalues for control signal of FIG. 7 Frequency Range Frequencies DutyCycle/On-Time Low f_(D) ≦ 10 Hz 10 Hz (75% V⁺ pulse, 25% V⁻ pulse) Mid10 Hz < f_(D) ≦ 16 Hz 75% High 16 Hz < f_(D) ≦ 100 Hz  75% @ 16 Hzf_(D) > 100 Hz 100% @ 100 Hz and above

In Table 3, the three frequency ranges can correspond, for example, tothe three frequency ranges shown in FIG. 4. To vary the magnitude of ahaptic effect, the magnitude of the pulse is varied proportionally tothe desired increase or decrease of magnitude. Haptic effects to beoutput in the high-frequency range cause a user to perceive that themagnitude and frequency change proportionally to any changes in thedesired frequency of the control signal shown in FIG. 7.

FIG. 8 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention. The control signal shown inFIG. 8 is a magnitude-sweep control signal (also referred to as a“mag-sweep” signal), which sweeps through a variety of magnitude valuesto cause a desired haptic effect to be output. The magnitude-sweepcontrol signal shown in FIG. 8 can cause a corresponding haptic effectto be output, or can be used to modulate another control signal (e.g., aperiodic signal such as the signals shown in FIG. 6 and FIG. 7).

A lead-in pulse is provided at the beginning of the control signal shownin FIG. 8, which improves response time of the haptic device beingcontrolled by the control signal. The duration t_(p) of the pulse isvaried as a function of the magnitude level of the beginning of theeffect (e.g., at a level desired to begin the impulse portion of thesignal, or the ramp-up portion of the signal). The smaller the magnitudelevels at the beginning of the effect, the shorter the duration t_(p) ofthe lead-in pulse and, similarly, the larger the magnitude levels at thebeginning of the signal, the longer the duration t_(p) of the lead-inpulse. The duration t_(p) of the pulse can be varied according to thepercentage of maximum control signal magnitude (e.g., the voltagemagnitude) that one desires to reach by the end of the pulse (e.g., atthe value indicated as “level” in FIG. 9 and denoted L in Table 4), asshown below in Table 4. TABLE 4 Duration t_(P) of the lead-in pulse inFIG. 8 |Maximum Voltage| % t_(P)  0% ≦ L ≦ 47% 0 47% ≦ L ≦ 70% 25 ms 70%≦ L ≦ 100% 50 ms

FIG. 9 is a diagram of a control signal used to control a haptic device,according to an embodiment of the invention. The control signal shown inFIG. 9 is a magnitude sweep signal, similar to the control signal shownin FIG. 8, but having both a lead-in pulse and a braking pulse. Becausethe braking pulse is a negative pulse, the control signal shown in FIG.9 can also be considered a bi-directional signal. The braking pulsegenerally is executed to cause a large change in voltage to stop theactuation of a haptic device (i.e., to stop a device from outputting ahaptic effect currently being output, such as stopping the rotation of arotational haptic device).

The braking pulse is of opposite polarity to the lead-in pulse, and therest of the signal. The duration t_(b) of the braking pulse varies as afunction of the magnitude level of the signal at the end of the effect(i.e., immediately prior to initiating the braking pulse). The smallerthe magnitude level at the end of the effect (i.e., at the end of thecontrol signal shown in FIG. 9), the shorter the duration of the brakingpulse that is required. Various lengths of possible braking pulses areshown below in Table 5 according to the corresponding percentage ofvoltage magnitude (e.g., the voltage magnitude) at the end of the signal(e.g., at the value indicated as “End level” in FIG. 10 and denoted ELin Table 5) shown in FIG. 9. TABLE 5 Duration t_(b) of the braking pulsein FIG. 9 |Maximum Voltage| % t_(b)  0% ≦ EL ≦ 47% 0 47% ≦ EL ≦ 70% 25ms 70% ≦ EL ≦ 100% 50 ms

FIG. 10 is a diagram of linearization of voltages of a haptic device,according to an embodiment of the invention. The actuator of the hapticdevice may require increased voltage over the desired voltage toinitiate and achieve the intended effects. FIG. 10 illustrates anexample of a linearization table for such an actuator. As shown in thelinearization table of FIG. 10, the percentage of motor voltage to beginmovement of the motor can be about twenty percent.

Each of the control signals described above in connection with FIGS.9-10 can be used with a haptic device that produces an audio output, inaddition to outputting a haptic effect, when driven by the controlsignal. As the control signal changes the rate of movement associatedwith such haptic devices, the audio output also can change. This canoccur, for example, by varying a magnitude, frequency, and/or pulse ofthe control signal. Thus, by using the control signal to vary the audiooutput produced by a haptic device, the control signal can cause a userto sense a change in the frequency in the overall effect and to perceivea change in the haptic effect output by the haptic device.

Additional information and examples regarding control signals accordingto one or more embodiments of the invention are illustrated below inTables 6-10. For example, Table 6 shows the number of availablecontroller input frequencies for different frequency ranges for thesmooth controller. Table 7 shows the number of available controllerinput frequencies for different frequency ranges for the strong andsharp controllers. Table 8 shows motor inputs for various frequencyranges for smooth, strong and sharp controllers. Table 9 shows desiredperceived frequency and perceived magnitude for various frequency rangesfor smooth, strong and sharp controllers. Table 10 shows actualacceleration frequency, perceived frequency and perceived magnitude forvarious frequency ranges for smooth, strong and sharp controllers. TABLE6 Controller input frequencies for the smooth controller # of available# of available Controller Frequency frequencies at 200 Hz frequencies at1 kHz Range Range sampling sampling Low <6.6 Hz 170 between (1 Hz 850between (1 Hz and 8 Hz) and 6.6 Hz) Transition 6.6-10 Hz 10 50 High10-100 Hz 19 90

TABLE 7 Controller input frequencies for the strong and sharpcontrollers # of available # of available Controller Frequencyfrequencies at frequencies at 1 kHz Range Range 200 Hz sampling samplingLow <10 Hz 180 between (1 Hz 900 between (1 Hz and 10 Hz) and 10 HzTransition 10-16 Hz  8 37 High 16-100 Hz 11 52

TABLE 8 Motor input for various frequency ranges for smooth, strong andsharp controllers Motor input: Motor input: Motor input: ControllerRange Frequency Range Smooth Strong Sharp Low <6.6 Hz Unidir PulseUnidir Pulse @ Bidir Pulse @ Smooth @ 6.6 Hz with 10 Hz with 75% 10 Hzwith <10 Hz Strong, 50% duty duty cycle 75% duty cycle Sharp cycleTransition 6.6-10 Hz Unidir Pulse at Unidir Pulse at Bidir Pulse atSmooth chosen input chosen input chosen input 10-16 Hz, frequency atfrequency at frequency at Strong Sharp 50% duty 75% duty cycle 75% dutycycle cycle High 10-100 Hz Unidir Pulse at Unidir Pulse at Bidir Pulseat Smooth chosen input chosen input chosen input 16-100 Hz, frequencyand frequency and frequency and Strong Sharp duty cycle duty cycle dutycycle increases from increases from increases from 50% to 100% 75% to100% 75% to 100%

TABLE 9 Desired perceived frequency and perceived magnitude for variousfrequency ranges for smooth, strong and sharp controllers ControllerFrequency Desired Perceived Desired Perceived Range Range FrequencyMagnitude Comments Low <6.6 Hz Smooth <10 Hz Strong, Sharp

Controller input frequency matches, actual and perceived outputfrequency. Perceived magnitude is variable with PWM Transition 6.6-10 HzSmooth 10-16 Hz Strong, Sharp

Controller input frequency creates actual frequency with envelope. (Ifcontinuous spinning, then input frequency does not match outputfrequency.) Average energy delivered is the same. High 10-100 Hz Smooth16-100 Hz, Strong Sharp

Controller input frequency creates continuous spinning where that doesnot match output frequency Average energy delivered is increasing.

TABLE 10 Actual acceleration frequency, perceived frequency andperceived magnitude for various frequency ranges for smooth, strong andsharp controllers Controller Input Actual Frequency AccelerationPerceived Perceived Controller Range Range frequency frequency MagnitudeLow <6.6 Hz Smooth Controller input Controller input Perceived <10 HzStrong, frequency frequency magnitude is Sharp matches, actual. matchesvariable with perceived. PWM Transition 6.6-10 Hz Controller input Userperceives Perceived Smooth frequency envelope magnitude is 10-16 Hzmatches frequency variable with Strong, Sharp envelope PWM. frequency.High 10-100 Hz Controller input Controller input As controller Smoothfrequency does frequency does input frequency 16-100 Hz, NOT match NOTmatch is increased, Strong Sharp actual perceived magnitude isfrequency. frequency. perceived to increase.

FIG. 11 is a diagram of parameters associated with a smooth effectaccording to an embodiment of the invention. FIG. 12 is a diagram ofparameters associated with a strong effect according to an embodiment ofthe invention. FIG. 13 is a diagram of parameters associated with asharp effect according to an embodiment of the invention.

A system and method for controlling audio output associated with hapticeffects are discussed. Specific embodiments have been described above inconnection with separately controlling multiple frequencies, eitherusing a single controller or using multiple controllers, each of themultiple controllers being equally associated with a frequency range.Additionally, other embodiments have been discussed in connection withcontrolling an audio output associated with a haptic effect in at leastone of the frequency ranges. Also, the audio effect can be changed, suchthat a user senses a change in frequency in the overall effect andperceives that change in a haptic effect; in some cases, this can occurabove the frequency range where a user can tactilely detect variationsin frequency (e.g., within a diminished-sensitivity region). Thus, asthe frequency of the audio effect is increased, the user perceives anincrease in a frequency of the haptic effect associated with the audioeffect, even where such an increase results in a change in hapticfrequencies within the diminished-sensitivity region. Similarly, as thefrequency of the audio effect is decreased, the user perceives adecrease in frequency of the corresponding haptic effect, even wheresuch changes result in variations of haptic effect frequencies, whichare undetectable to a user (e.g., within the diminished-sensitivityregion).

It will be appreciated, however, that embodiments of the invention canbe in other specific forms without departing from the spirit oressential characteristics thereof. For example, while some embodimentshave been described in the context of periodic or magnitude sweepcontrol signals for causing haptic effects, any suitable signal can beused. Also, although control signals have been described as square-wavesor PWM signals having square-wave-like shapes, other pulse shapes can beused. Additionally, although a specific reference has been made todevices configured to output periodic haptic effect (e.g., rotatinghaptic devices such spinning mass motors, etc.), any type of hapticdevice capable of outputting haptic effects associated with an audiooutput can be used according to one or more embodiments of theinvention.

The presently disclosed embodiments are, therefore, considered in allrespects to be illustrative and not restrictive.

1. A processor-readable medium comprising code representing instructions to cause a processor to: send a signal configured to cause a haptic effect and an audio effect to be output substantially concurrently, the haptic effect having a frequency and the audio effect having a frequency different from the frequency of the haptic effect, the signal being further configured to vary at least one of the frequency of the haptic effect and the frequency of the audio effect while maintaining substantially constant an average energy of the haptic effect.
 2. The processor-readable medium of claim 1, wherein the signal is further configured to cause the frequency of the audio effect to vary while causing the frequency of the haptic effect to remain substantially constant.
 3. The processor-readable medium of claim 1, wherein the signal includes a plurality of pulses, the signal being configured to cause a frequency of the plurality of pulses to vary while causing a magnitude of the plurality of pulses to vary inversely.
 4. The processor-readable medium of claim 1, the frequency of the haptic effect being a first frequency of the haptic effect, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a second frequency different from the first frequency of the haptic effect.
 5. The processor-readable medium of claim 1, the frequency of the haptic effect being a first frequency of the haptic effect, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a second frequency different from the first frequency of the haptic effect, the second frequency being higher than the first frequency.
 6. The processor-readable medium of claim 1, wherein the frequency of the audio effect is higher than the frequency of the haptic effect.
 7. The processor-readable medium of claim 1, wherein the haptic effect has a frequency from a frequency range from a plurality of frequency ranges, each frequency range from the plurality of frequency ranges being uniquely associated with a control scheme.
 8. An apparatus, comprising: a controller configured to output a control signal, the control signal being configured to cause a haptic effect and an audio effect to be output substantially concurrently, the haptic effect having a frequency and the audio effect having a frequency different from the frequency of the haptic effect, the control signal being further configured to vary at least one of the frequency of the haptic effect and the frequency of the audio effect while maintaining substantially constant an average energy of the haptic effect; and an interface component coupled to the controller and configured to be coupled to a component external to the controller, the interface component configured to provide a haptic instruction to the component based at least partially on the control signal.
 9. The apparatus of claim 8, wherein the control signal is further configured to cause the frequency of the audio effect to vary while causing the frequency of the haptic effect to remain substantially constant.
 10. The apparatus of claim 8, wherein the control signal includes a plurality of pulses, the controller being configured to cause a frequency of the plurality of pulses to vary while causing a magnitude of the plurality of pulses to vary inversely.
 11. The apparatus of claim 8, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect.
 12. The apparatus of claim 8, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect, the perceived frequency being higher than the frequency of the haptic effect.
 13. The apparatus of claim 8, wherein the control signal is configured to cause a plurality of haptic effects including the haptic effect, each of the plurality of haptic effects having a frequency within at least two frequency ranges from a plurality of frequency ranges, the control signal being configured to cause the haptic effect to be output having a frequency in an intermediate frequency range from the plurality of frequency ranges, the intermediate frequency range being between the at least two frequency ranges.
 14. The apparatus of claim 8, further comprising: a haptic device coupled to the controller, the haptic device being configured to receive the control signal and to output the haptic effect and the audio effect.
 15. The apparatus of claim 8, further comprising: a haptic device coupled to the controller, the haptic device including an actuator, the haptic device being configured to receive the control signal and to output the haptic effect and the audio effect via the actuator.
 16. The apparatus of claim 8, further comprising: a haptic device coupled to the controller, the haptic device including a actuator and an audio output device substantially collocated with the actuator and to receive the control signal, the haptic device being configured to output the haptic effect via the actuator, the haptic device being configured to output the audio effect via the audio output device.
 17. The apparatus of claim 8, further comprising: a plurality of controllers including the controller, each controller from the plurality of controllers being associated with a frequency range from a plurality of frequency ranges, each controller from the plurality of controllers being configured to output an associated control signal, the associated control signal output by each controller from the plurality of controllers being configured to cause a haptic effect to be output having a frequency within the frequency range associated with that controller, at least one controller from the plurality of controllers being configured to output the control signal configured to cause the haptic effect and the audio effect to be output substantially concurrently.
 18. The apparatus of claim 8, further comprising: a resonant vibrotactile haptic device coupled to the controller, the resonant vibrotactile haptic device being configured to output the haptic effect having a frequency within a pre-determined operational frequency range, the pre-determined operational frequency range having a frequency associated with a resonant mode of the resonant vibrotactile haptic device.
 19. The apparatus of claim 8, wherein the control signal includes a plurality of pulses, the control signal being configured to cause the haptic effect to be output at a desired output frequency, each pulse from the plurality of pulses having a width associated with the desired output frequency.
 20. The apparatus of claim 8, further comprising: a plurality of controllers including the controller, each controller from the plurality of controllers being configured to output a control signal from a plurality of control signals, each control signal from the plurality of control signals being uniquely associated with a frequency range from a plurality of frequency ranges and being configured to cause a haptic effect to be output, a first control signal from the plurality of control signals being output by a first controller, the first control signal being configured to cause the haptic effect to be output substantially concurrently with the audio effect, the audio effect being configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect.
 21. The apparatus of claim 8, further comprising: a plurality of controllers including the controller, each controller from the plurality of controllers being configured to output a control signal from a plurality of control signals, each control signal from the plurality of control signals being uniquely associated with a frequency range from a plurality of frequency ranges and being configured to cause a haptic effect to be output, a first control signal from the plurality of control signals being output by a first controller, the first control signal being configured to cause the haptic effect to be output substantially concurrently with the audio effect, the audio effect being configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect; and a haptic device coupled to the plurality of controllers, the haptic device being configured to output a plurality of haptic effects associated with the plurality of control signals, the plurality of haptic effects including the haptic effect.
 22. The apparatus of claim 8, further comprising: a plurality of controllers including the controller, each controller from the plurality of controllers being configured to output a control signal from a plurality of control signals, each control signal from the plurality of control signals being uniquely associated with a frequency range from a plurality of frequency ranges and being configured to cause a haptic effect to be output, a first control signal from the plurality of control signals being output by a first controller, the first control signal being configured to cause the haptic effect to be output substantially concurrently with the audio effect, the audio effect being configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect; and a plurality of haptic devices, each haptic device from the plurality of haptic devices being uniquely associated with a controller from the plurality of controllers, each haptic device from the plurality of haptic devices being configured to output the haptic effect caused by the control signal output by the associated controller.
 23. A method, comprising: outputting a haptic effect at least partially based on a control instruction; and outputting an audio effect substantially concurrently with a haptic effect at least partially based on the control instruction, the haptic effect having a frequency, the audio effect having a frequency different from the frequency of the haptic effect, the audio effect and the haptic effect being output by a common device; varying the frequency of at least one of the frequency of the haptic effect and the frequency of the audio effect while maintaining substantially constant an average energy of the haptic effect.
 24. The method of claim 23, wherein the frequency of the audio effect is varied while the frequency of the haptic effect is maintained substantially constant.
 25. The method of claim 23, wherein outputting the haptic effect includes: varying a frequency of pulses configured to cause the haptic effect to be output while inversely varying a magnitude of the pulses configured to output the haptic effect.
 26. The method of claim 23, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect.
 27. The method of claim 23, wherein the audio effect is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect, the perceived frequency of the haptic effect being higher than the frequency of the haptic effect.
 28. The method of claim 23, further comprising: outputting a plurality of haptic effects, the plurality of haptic effects including the haptic effect, each haptic effect from the plurality of haptic effects having a different corresponding frequency, each corresponding frequency being within a frequency range from a plurality of frequency ranges.
 29. A method, comprising: receiving an output instruction; and outputting a control signal based on the received output instruction, the control signal being configured to cause a haptic effect and an audio effect to be output substantially concurrently, the haptic effect having a frequency and the audio effect having a frequency different from the frequency of the haptic effect, the control signal being configured to vary at least one of the frequency of the haptic effect and the frequency of the audio effect while maintaining substantially constant an average energy of the haptic effect.
 30. The method of claim 29, wherein the control signal is configured to cause the frequency of the audio effect to vary while causing the frequency of the haptic effect to remain substantially constant.
 31. The method of claim 29, wherein the control signal includes a plurality of pulses, configured to cause a frequency of the plurality of pulses to vary while causing a magnitude of the plurality of pulses to vary inversely.
 32. The method of claim 29, wherein the control signal is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect.
 33. The method of claim 29, wherein the control signal is configured to cause a user to perceive the haptic effect as having a perceived frequency different from the frequency of the haptic effect, the perceived frequency of the haptic effect being higher than the frequency of the haptic effect.
 34. The method of claim 29, further comprising: outputting a plurality of control signals including the control signal, the plurality of control signals being at least partially based on the received output instruction, each control signal from the plurality of control signals being uniquely associated with a frequency range from a plurality of frequency ranges, at least one control signal from the plurality of control signals being configured to cause the haptic effect.
 35. The method of 29, wherein the control signal is configured to output the haptic effect and the audio effect via an actuator of a haptic device.
 36. The method of 29, wherein the control signal is configured to output the haptic effect via an actuator of a haptic device, the control signal being configured to cause the audio effect to be output by an audio output device of the haptic device, the audio output device being substantially collocated with the actuator.
 37. A method, comprising: receiving an output instruction; and outputting a control signal based on the received output instruction, the control signal being configured to cause a haptic effect and an audio effect to be output substantially concurrently via an actuator of a haptic device, the haptic effect having a frequency and the audio effect having a frequency different from the frequency of the haptic effect, the audio effect being configured to change a perceived frequency of the haptic effect. 